Computing system for extraction of textual elements from a document

ABSTRACT

Described herein are various technologies pertaining to text extraction from a document. A computing device receives the document. The document comprises computer-readable text and a layout, wherein the layout defines positions of the computer-readable text within a two-dimensional area represented by the document. Responsive to receiving the document, the computing device identifies at least one textual element in the computer-readable text based upon spatial factors between portions of the computer-readable text and contextual relationships between the portions of the computer-readable text. The computing device then outputs the at least one textual element.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/358,271, filed on Mar. 19, 2019, and entitled “COMPUTING SYSTEM FOREXTRACTION OF TEXTUAL ELEMENTS FROM A DOCUMENT”, the entirety of whichis incorporated herein by reference.

FIELD

This disclosure relates to computer-implemented text and characterrecognition systems and methods.

BACKGROUND

A computer-readable document comprises computer-readable text and alayout. The layout defines positions of the computer-readable textwithin a two-dimensional area represented by the document. Such adocument may, for example, be a semi-structured document. The documentmay thus serve as a digital representation of a physical copy of thedocument while at the same time retaining certain characteristics (e.g.,length, width) of the physical copy.

As documents comprise computer-readable text, a computing device mayperform a search over computer-readable text in a document in order toidentify and extract relevant textual elements in the text. Thecomputing device may then save the textual elements in a format that issuitable for further data processing (e.g., as part of a data structure,as part of a spreadsheet, as an entry in a database). In oneconventional approach for identifying and extracting textual elementsfrom a document, the computing device performs regular expressionmatching in order to identify and extract the textual elements. Inanother conventional approach, the computing device utilizes a templatein order to identify and extract the textual elements from the document.The template is based upon expected positions of the portions of thecomputer-readable text within the document.

Conventional text extraction techniques for documents suffer fromvarious deficiencies. First, conventional techniques tend to beinflexible with respect to changes in content of a document. Forinstance, if a textual element in a type of document is changed from afirst word to a second word and a corresponding regular expression isnot modified to incorporate the change, the computing device thatexecutes the regular expression may be unable to properly match thetextual element in the text of the document. Second, conventionaltechniques also tend to be inflexible with respect to changes (bothmajor and minor) in a layout of a document. For instance, if a positionof the textual element in the document is changed and a correspondingtemplate for the document is not modified to incorporate the change, thecomputing device that is to match the template to the text in thedocument may be unable to correctly capture the textual element in thetext of the document. Generating new regular expressions and/or newtemplates each time a document is changed can be burdensome on resourcesof the computing device.

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

Described herein are various technologies pertaining to extraction oftextual elements from a document. With more specificity, describedherein is a textual extraction application that is configured toidentify at least one textual element in a document based upon spatialfactors between portions of computer-readable text in the document andcontextual relationships between the portions of the computer-readabletext.

In operation, a computing device that executes the textual extractionapplication receives a document comprising computer-readable text and alayout. The computer-readable text may include letters, numbers,punctuation, and/or mathematical symbols. The layout defines positionsof the computer-readable text within a two-dimensional area representedby the document. The document may have a defined type, wherein thedefined type is indicative of a purpose of the document. In an example,a defined type of a document may be an educational transcript, and assuch, computer-readable text of the educational transcript may beindicative of classes taken by a student, credit hours received by thestudents for the classes, and grades that the student received in theclasses. In a further example, portions of the computer-readable textand/or the layout of the document may not have been encounteredpreviously by the textual extraction application.

Responsive to receiving the document, the textual extraction applicationidentifies at least one textual element in the computer-readable textbased upon spatial factors between portions of the computer-readabletext in the document and contextual relationships between the portionsof the computer-readable text. The spatial factors may include distancesbetween the portions of the computer-readable text, angles between theportions of the computer-readable text and an axis of the document,and/or orderings between the portions of the computer-readable text. Thetextual extract application may calculate the spatial factors based uponthe positions of the computer-readable text within the document. Thecontextual relationships are determined via at least onecomputer-implemented model. Exemplary contextual relationships includesource to object, object to use, person to location, whole to part,and/or type to subtype.

In order to identify the at least one textual element, the textualextraction application provides the computer-readable text and thepositions of the computer-readable text in the document as input to theat least one computer-implemented model. The at least onecomputer-implemented model has been generated based upon a plurality ofdocuments having the defined type of the document, wherein at least somedocuments in the plurality of documents have varying layouts, varyingareas, varying portions of computer-readable text, varying font types,varying lengths and widths, and/or varying font sizes. The at least onecomputer-implemented model outputs, based upon the input, a plurality oftextual elements in the computer-readable text and scores assigned tothe plurality of textual elements. The at least one textual element isincluded in the plurality of textual elements. The textual extractionapplication identifies the at least one textual element based on a scorein the scores. The score is indicative of a likelihood that the at leastone textual element represents relevant content in the document basedupon defined criteria for the defined type of the document. The definedcriteria may be received by the textual extraction application via inputby a user of the computing device. In an example where the document isan educational transcript, the defined criteria may indicate thatidentifiers for mathematical classes taken by the student and grades forthe mathematical classes are to be extracted from the educationaltranscript.

Responsive to identifying the at least one textual element, the textualextraction application outputs the at least one textual element. In anexample, outputting the at least one textual element comprises causingthe computing device to present the at least one textual element on adisplay. In another example, outputting the at least one textual elementcomprises causing the computing device to store the at least one textualelement in a data structure that is suitable for further dataprocessing.

The above-described technologies present various advantages overconventional computer-implemented text extraction techniques. First, theabove-described technologies enable extraction of textual elements froma new document of a defined type that has different content (e.g.,different identifiers to refer to the same content) than a previousdocument of the defined type, without having to resort to generating newregular expressions to account for the different content. Second, theabove-described technologies enable extraction of textual elements ofdocuments of the defined type that have varied layouts, without havingto resort to generating new templates to account for the varied layouts.Thus, the above-described technologies reduce usage of computingresources of computing devices.

The above summary presents a simplified summary in order to provide abasic understanding of some aspects of the systems and/or methodsdiscussed herein. This summary is not an extensive overview of thesystems and/or methods discussed herein. It is not intended to identifykey/critical elements or to delineate the scope of such systems and/ormethods. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an exemplary computing devicethat facilitates extracting textual elements from computer-readable textof a document.

FIG. 2 illustrates an exemplary document.

FIG. 3 depicts an exemplary graphical user interface (GUI).

FIG. 4 depicts another exemplary GUI.

FIG. 5 depicts yet another exemplary GUI.

FIG. 6 depicts exemplary textual elements.

FIG. 7 is a flow diagram that illustrates an exemplary methodologyexecuted by a computing device for generating a computer-implementedmodel.

FIG. 8 is a flow diagram that illustrates an exemplary methodologyexecuted by a computing device for extracting textual elements fromcomputer-readable text of a document.

FIG. 9 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to extraction of textual elements fromcomputer-readable text of documents are now described with reference tothe drawings, wherein like reference numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more aspects. It may be evident,however, that such aspect(s) may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to facilitate describing one or moreaspects. Further, it is to be understood that functionality that isdescribed as being carried out by certain system components may beperformed by multiple components. Similarly, for instance, a componentmay be configured to perform functionality that is described as beingcarried out by multiple components.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a,”“an,” and “the” as used in this application and the appended claimsshould generally be construed to mean “one or more” unless specifiedotherwise or clear from the context to be directed to a singular form.In additive, the term “including” should be construed as “including,without limitation.”

Further, as used herein, the terms “component,” “application,” and“system” are intended to encompass computer-readable data storage thatis configured with computer-executable instructions that cause certainfunctionality to be performed when executed by a processor. Thecomputer-executable instructions may include a routine, a function, orthe like. It is also to be understood that a component or system may belocalized on a single device or distributed across several devices.Further, as used herein, the term “exemplary” is intended to meanserving as an illustration or example of something, and is not intendedto indicate a preference.

With reference to FIG. 1, an exemplary computing device 100 thatfacilitates extracting textual elements from computer-readable text of adocument is illustrated. The computing system 100 includes a processor102 and memory 104, wherein the memory 104 has a textual extractionapplication 106 loaded therein. As will be described in greater detailbelow, the textual extraction application 106 (when executed by theprocessor 102) is configured to extract textual elements fromcomputer-readable text of documents based upon spatial factors betweendifferent portions of the computer-readable text in the documents andcontextual relationships between the different portions of thecomputer-readable text in the documents. In an embodiment, the textualextraction application 106 may also be configured to present a graphicaluser interface (GUI) that facilitates defining criteria for definedtypes of documents that is to be utilized by the textual extractionapplication 106 in extracting textual elements from the documents.

In an embodiment, the computing device 100 may be operated by a user. Assuch, the computing device 100 may comprise a display 108, whereupongraphical features 110 may be presented thereon. For instance, one ormore GUIs (described below) may be presented on the display 108 as partof the graphical features 110.

Furthermore, the computing device 100 may additionally include inputcomponents 112 suitable for data input. For instance, the inputcomponents 112 may include a mouse, a keyboard, a touchscreen, atrackpad, a scroll wheel, a microphone, a camera, and/or a video camera.

The computing device 100 may include a data store 114. The data store114 may store a document image 116. The document image 116 fails toinclude computer-readable text. More specifically, the document image116 includes pixels that are indicative of text when presented on thedisplay 108, but the text itself is not computer-readable and hence notsearchable by the computing device 100. Exemplary file formats for thedocument image 116 include, but are not limited to, a non-searchableportable document format (PDF), a joint photographic experts group(JPEG), a JPEG file interchange format (JFIF), a tagged imaged fileformat (TIFF), a graphics interchange format (GIF), a windows bitmap(BMP), and/or a portable network graphics (PNG). Although the data store114 has been depicted and described as storing a single document image,it is to be understood that the data store 114 may store many differentdocument images. Moreover, the document images may have defined types(described in greater detail below).

In an embodiment, the computing device 100 may be in communication witha scanner (not shown). The scanner may generate the document image 116by scanning a physical copy of a document.

The data store 114 also stores a document 118. The document 118comprises computer-readable text (i.e., text that is searchable by thecomputing device 100) and a layout. The computer-readable text mayinclude combinations of American Standard Code for InformationInterchange (ASCII) characters and/or combinations of Unicodecharacters. For instance, the computer-readable text may includeletters, numbers, punctuation, and/or mathematical symbols.

The layout defines positions of the computer-readable text within atwo-dimensional area represented by the document 118. Thus, the document118 has a length and a width. In a non-limiting example, thetwo-dimensional area may correspond to an A4 paper size, a letter papersize, or a legal paper size.

In an embodiment, the document 118 may be a tabular document such thatthe computer-readable text is arranged within one or more tables in thedocument 118. Thus, in the embodiment, the layout of the document 118may define positions of the computer-readable text within the one ormore tables.

It is to be understood that the document 118 may comprise one or morepages. It is also to be understood that different portions of thecomputer-readable text within the document 118 may be presented indifferent fonts (e.g., Times New Roman, Courier New) and/or differentfont sizes (e.g., 12-point font, 14-point font). For instance, a firstportion of the computer-readable text may be presented in a first font,whereas a second portion of the computer-readable text may be presentedin a second font. The first portion may be presented in 12-point font,whereas the second portion may be presented in 14-point font.Furthermore, it is to be understood that some portions of thecomputer-readable text within the document 118 may have typographicalemphasis applied thereto. The typographical emphasis may include bold,underline, and/or italics.

The document 118 may have a defined type, wherein the defined type isindicative of a purpose of the document 118, and it should be understoodthat various documents with a specified purpose have similar relevantcontent often in similar spatial formats. In an example, the definedtype may be an educational transcript that conveys information relatingto grades received by a student for classes completed by the student. Inanother example, the defined type may be a taxation form that includesfinancial information of an entity that is used in determining taxesincurred by the entity. In yet another example, the defined type may bean invoice for goods or services. In a further example, the defined typemay be a medical record. In an additional example, the defined type maybe a personnel record. Other defined types may include human resourcerelated documents, financial documents, such as documents related toinsurance and mortgages, business cards, identification documents, suchas drivers' licenses or visa documents, ballot papers, trade documents,bills of lading, and/or bank statements. In such examples, there will besimilar context and text, such as capital letters standing alone, e.g.,A, B, C, D, or F, or with a plus or minus sign for a transcript, andnumerical values near a side or bottom of a document for an invoice ortax form. While there are similarities in these documents, there arealso substantial variations, e.g., in location and how the content ispresented. These variations make errors in automatic reading andextraction of relevant information from such documents by a computingdevice a significant problem, which is addressed by the featuresdisclosed herein.

In an embodiment, the computing device 100 (or another computing device)may generate the document 118 from the document image 116 by applying anoptical character recognition (OCR) process to the document image 116.Exemplary file formats for the document 118 include, but are not limitedto, a searchable PDF and/or a document format, such as .doc or .docx.

Although the data store 114 has been depicted and described as storing asingle document (the document 118), it is to be understood that the datastore 114 may store many different documents having varying areas,layouts, computer-readable text, fonts, font sizes, and/or typographicalemphasis. Moreover, the many different documents may be of differentdefined types.

The data store 114 may further store a computer-implemented model 120that may be executed by the textual extraction application 106 in orderto extract relevant textual elements from the document 118. In anembodiment, the computer-implemented model 120 may be acomputer-implemented machine learning model. The computer-implementedmodel 120 is generated based upon a plurality of documents having adefined type, wherein characteristics (described in greater detailbelow) of at least some documents in the plurality of documents vary. Ingeneral, the computer-implemented model 120 is configured to take, asinput, computer-readable text from a document (e.g., the document 118)having a defined type and positions of the computer-readable text withinthe document. The computer-implemented model 120 is configured tooutput, based upon the input, a plurality of textual elements from thecomputer-readable text and a score that is assigned to each textualelement in the plurality of textual elements. Each score in the scoresis indicative of a likelihood that each textual element in the pluralityof textual elements represents relevant content in the document basedupon defined criteria (described in greater detail below) for thedefined type of the document.

In an embodiment, the computer-implemented model 120 may be or include apredictive model. The predictive model may be or include a continuousbag-of words model, a skip-gram model, or a weighed n-gram differencesmodel. In another embodiment, the computer-implemented model 120 may beor include a count-based model, such as a Latent Semantic Analysis (LSA)model. In an embodiment, the computer-implemented model 120 mayincorporate t-distributed stochastic neighbor embedding (t-SNE)techniques.

In an embodiment, the computer-implemented model 120 may be or include aBayesian hierarchical model, a random forest model, or a neural networkmodel. When the computer-implemented model 120 is a neural networkmodel, the neural network model may be or include an artificial neuralnetwork (ANN) model, a deep neural network (DNN) model, a convolutionalneural network (CNN) model, a recurrent neural network (RNN) model, oranother suitable neural network model.

In an example, the computer-implemented model 120 may comprise nodes andedges that couple nodes in the computer-implemented model 120. Each edgeis assigned a learned weight, wherein the learned weight can be learnedusing a supervised, semi-supervised, or unsupervised learning procedure.Accordingly, for instance, a learned weight assigned to an edge can beinfluenced by characteristics of a plurality of documents of a definedtype. For instance, the characteristics may include computer-readabletext of the plurality of documents, positions of the computer-readabletext within the plurality of documents, areas of the plurality ofdocuments, lengths and widths of the plurality of documents, fonts ofthe plurality of documents, font sizes of the plurality of documents,and/or typographical emphasis within the plurality of documents. Thecomputer-implemented model 120 may take, as input, computer-readabletext from a document (e.g., the document 118) and positions of thecomputer-readable text within the document. The computer-implementedmodel 120 is configured to output, based upon the input and the learnedweights, a plurality of textual elements and scores assigned to eachtextual element in the plurality of textual elements. Each score in thescores is indicative of a likelihood that each textual element in theplurality of textual elements represents relevant content in thedocument based upon defined criteria for the defined type of document.

Although the data store 114 has been depicted and described as storing asingle computer-implemented model (the computer-implemented model 120),it is to be understood that the data store 114 may store many differentcomputer-implemented models (and hence the computer-implemented model120 may comprise multiple computer-implemented models). In an example,the data store 114 may include a first computer-implemented model thatis configured to identify textual elements in the document 118 basedupon spatial factors between portions of the computer-readable text inthe document 118 and a second computer-implemented model that isconfigured to identify the textual elements in the document 118 basedupon contextual relationships between the portions of thecomputer-readable text. The textual extraction application 106 mayexecute both the first computer-implemented model and the secondcomputer-implemented model in order to identify textual elements in thedocument 118 that represent relevant content in the document 118.

The data store 114 may additionally store extracted textual elements 122that the textual extraction application 106 has extracted from thedocument 118 (or from other documents). The extracted textual elements122 may be stored in a data structure that is conducive to further dataprocessing. For instance, the extracted textual elements 122 may bestored in a spreadsheet (e.g., an eXtensible Markup Language (XML) basedspreadsheet) or in a comma separated value (CSV) structure. Theextracted textual elements 122 may also be stored as one or more entriesin a database.

Although the document image 116, the document 118, thecomputer-implemented model 120, and the extracted textual elements 122are described and shown as being stored in the data store 114, it is tobe understood that the document image 116, the document 118, thecomputer-implemented model 120, and/or the extracted textual elements122 may also be retained in the memory 104 of the computing device 100.

Operation of the computing device 100 is now set forth. Prior toidentifying at least one textual element from computer-readable text ofa document having a defined type, the computing device 100 (or anothercomputing device) generates the computer-implemented model 120 from aplurality of documents (i.e., a corpus), each document in the pluralityof documents having a defined type (e.g., educational transcript). Eachdocument in the plurality of documents comprises computer-readable textand a layout that defines positions of the computer-readable text withina two-dimensional area represented by each document. Characteristics ofsome documents in the plurality of documents may vary. For instance, thecharacteristics may include varying portions of computer-readable text,varying positions of the computer-readable text within each document inthe plurality of documents, varying typographical emphasis of portionsof the computer-readable text, varying areas, varying lengths andwidths, varying font sizes of the portions of the computer-readabletext, and/or varying font sizes of the portions of the computer-readabletext. In a specific example, a first document in the plurality ofdocuments may have a textual element located at a first position withinthe first document, while a second document in the plurality ofdocuments may have the (same) textual element located at a secondposition within the second document. The computing device 100 may repeatthis process for documents having different defined types in order togenerate many different computer-implemented models for the differentdefined types of documents.

The computing device 100 may also receive an indication of definedcriteria for defined type of document. For instance, the computingdevice 100 may receive the indication of the defined criteria as inputfrom a user of the computing device 100. The textual extractionapplication 106 may utilize the defined criteria in order to identifyrelevant content in a document of the defined type. More specifically,the defined criteria may be incorporated into the computer-implementedmodel 120. In an example where the type of the document is an educationtranscript, the defined criteria may be an indication of a type ofcourse (e.g., mathematics courses having an identifier of “MATHXXX”,where XXX is a number) taken by a student and grades received by thestudent in the type of course. As will be described in greater detailbelow, the textual extraction application 106 may utilize the definedcriteria to identity a textual element in the document 118 even when thetextual element is not identical to the defined criteria. For example,the textual extraction application 106 may identify mathematics courseshaving an identifier of “MTHYYYY”, where YYYY is a number, even when thetextual extraction application 106 has not been explicitly configured toidentify mathematics courses having the aforementioned identifier.

The textual extraction application 106 receives the document 118. In anexample, the computing device 100 may receive the document 118 from asecond computing device that is in network communication with thecomputing device 100. In another example, the computing device mayaccess the document 118 from the data store 114. As described above, thedocument 118 comprises computer-readable text and a layout. The layoutdefines positions of the computer-readable text within a two-dimensionalarea represented by the document 118.

The textual extraction application 106 identifies at least one textualelement in the computer-readable text based upon spatial factors betweenportions of the computer-readable text and contextual relationshipsbetween the portions of the computer-readable text. For instance, thetextual extraction application 106 may identify the at least one textualelement responsive to receiving the document 118. The spatial factorsmay be present in a variety of forms. In an example, thecomputer-readable text may include a first textual element and a secondtextual element. The first textual element may be a first word locatedat a first position within the document 118, whereas the second textualelement may be a second word located at a second position within thedocument 118. The spatial factors may then include a distance betweenthe first textual element and the second textual element. The distancemay be expressed in a metric unit (e.g., centimeters, millimeters) or asa number of pixels from the first textual element to the second textualelement. In an example, the distance may be 0.01 to 20 mm. For instance,the distance may be 0.1 to 10 mm, or 1 to 5 mm.

The spatial factors may additionally include an angle between the firstlocation of the first textual element, the second location of the secondtextual element, and an axis of the document. For instance, the axis maybe a horizontal axis (i.e., a top side of the document 118 or a bottomside of the document 118) of the document 118 or a vertical axis of thedocument 118 (i.e., a left side of the document 118 or a right side ofthe document 118). In an example, the angle may range from 0 to 180°.For instance, the angle may be 15 to 80°, or 30 to 45°. When the atleast one textual element further includes a third textual element at athird location within the document 118, the angle may be calculatedbased upon an angle within a triangle formed between the first locationof the first textual element, the second location of the second textualelement, and the third location of the third textual element.

The spatial factors may also include an ordering between the firsttextual element and the second textual element within the document 118.More specifically, the ordering may be an indication of whether thetextual element or the second textual element occurs first in thedocument 118. The ordering may account for a read order (e.g., left toright, right to left, top to bottom, bottom to top) of a language inwhich the document 118 is written. The spatial factors may also includea number of textual elements that occur between the first textualelement and the second textual element.

As mentioned previously, the textual extraction application 106 mayfurther identify the at least one textual element based upon contextualrelationships between the portions of the computer-readable text. Thecontextual relationships may be determined via the computer-implementedmodel 120. Exemplary contextual relationships include, but are notlimited to, source to object, object to use, person to location, wholeto part, and/or type to subtype.

The textual extraction application 106 identifies the at least onetextual element by providing the computer-readable text of the document118 and the positions of the computer-readable text within the document118 as input to the computer-implemented model 120. In an embodiment,the textual extraction application 106 calculates the spatial factorsdescribed above and provides the calculated spatial factors as input tothe computer-implemented model 120 in addition to providing thecomputer-readable text and the positions of the computer-readable textwithin the document 118.

In an embodiment, the textual extraction application 106 may identifythe at least one textual element in the computer-readable text basedupon types of typographical emphasis of portions of thecomputer-readable text. For instance, a textual element may be scoredhigher if the textual element is bolded within the document 118. Thetextual extraction application 106 may provide indications of portionsof the computer-readable text that have the typographical emphasis tothe computer-implemented model 120. The plurality of textual elementsand the scores output by the computer-implemented model 120 may thus befurther based upon the typographical emphasis of the portions ofcomputer-readable text.

In an embodiment, the textual extraction application 106 may identifythe at least one textual element in the computer-readable text basedupon font types and/or font sizes of the computer-readable text in thedocument 118. The textual extraction application 106 may provideindications of the font types and/or the font sizes of thecomputer-readable text to the computer-implemented model 120. Theplurality of textual elements and the scores output by thecomputer-implemented model 120 may thus be further based upon the fonttypes and/or font sizes of the computer-readable text.

It is to be understood that the at least one textual element identifiedby the textual extraction application 106 may vary in length and/ortype. In an example, the at least one textual element may include afirst textual element and a second textual element. The first textualelement may be a word in the computer-readable text of the document 118,while the second textual element may be a number in thecomputer-readable text of the document 118. In another example, thefirst textual element may be indicative of an identifier for the definedcriteria that is found within the computer-readable text of the document118, while the second textual element may be a word that meets thedefined criteria. In yet another example, the first textual element mayinclude a first word and a second word, while the second textual elementmay include a third word.

In an embodiment, the textual extraction application 106 may calculatestring metrics for portions of the computer-readable text in thedocument 118. For instance, the string metrics may include Levenshteindistance, Damerau-Levenshtein distance, longest common subsequence (LCS)distance, Hamming distance, and/or Jaro distance. The textual extractionapplication 106 may further identify the at least one textual elementbased upon the string metrics.

Responsive to identifying the at least one textual element, the textualextraction application 106 outputs the at least one textual element. Inan example, the textual extraction application 106 may output the atleast one textual element by presenting the at least one textual elementas part of the graphical features 110 presented on the display 108 ofthe computing device 100. In another example, the textual extractionapplication 106 may output the at least one textual element by storingthe at least one textual element in a data structure that is conduciveto further data processing. For instance, the textual extractionapplication 106 may cause the at least one textual element to be storedin an eXtensible Markup Language (XML) file (e.g., an XML-basedspreadsheet), in a comma separated value (CSV) file, or as an entry in adatabase. The textual extraction application 106 may store the at leastone textual element from the document 118 as part of the extractedtextual elements 122 stored in the data store 114.

Although the above-described processes have been described asidentifying and extracting textual elements in a single document, it isto be understood that the textual extraction application 106 mayidentify and extract textual elements from many different documents.Moreover, the textual extraction application 106 may identify andextract textual elements from documents of different defined types. Itis to be understood that the textual extraction application 106 (oranother application) may update the computer-implemented model 120 whenfurther documents of the defined type are received by the computingdevice 100.

In an embodiment, the textual extraction application 106 may beconfigured to classify a document of an unknown type. The textualextraction application 106 may provide the computer-implemented textand/or positions of the computer-readable text within the document to acomputer-implemented model (e.g., one of the computer-implemented modelsdescribed above). The computer-implemented model may then output alabel, wherein the label is indicative of a type of the document.

The textual extraction application 106 may be advantageous in scenariosin which the document 118 is an updated version of a second (prior)document. The second document may comprise second computer-readable textand have a second layout defining second positions of the secondcomputer-readable text within a second two-dimensional area representedby the second document. The second computer-readable text and/or thesecond layout may vary from the computer-readable text and the layout ofthe document 118, respectively. Through execution of the above-describedprocesses, the textual extraction application 106 can correctly extractat least one textual element from the document 118 despite the fact thatthe textual extraction application 106 has not been explicitlyconfigured to extract textual elements from the document 118.

It is to be understood that the aforementioned processes described abovemay be performed by more than one computing device. For instance, aserver textual extraction application may execute on a server computingdevice and a client textual extraction application may execute on aclient computing device that is in network communication with the servercomputing device. The client computing device may present an interfacethat enables a user to define criteria (“the defined criteria”) fordocuments of a defined type, while the server computing device mayexecute computer-implemented models in order to identify the at leastone textual element. The server computing device may then, for example,cause the at least one textual element to be stored as an entry in adatabase. The server computing device may also transmit the at least onetextual element to the client computing device, whereupon the clientcomputing device can present the at least one textual element on adisplay of the client computing device.

Referring now to FIG. 2, an exemplary document 200 that can be presentedon a display of a computing device (e.g., as part of the graphicalfeatures 110 presented the display 108 of the computing device 100) isillustrated. In an example, the document 200 may be or include thedocument 118. As shown in FIG. 2, the document 200 has a defined type,that is, the document 200 is an educational transcript. As such, thedocument 200 includes a name of a student (“Joe A. Smith”), a uniqueidentifier for the student (“123456”), an address of the student (“1231^(st) Street, APT. 101, Greentown, OH, 44233”), a list of classescompleted by the student during Winter 2019 (“Calculus I, Physics II,Chemistry I, and Biology I”), identifiers for the classes (“MATH101,PHYS201, CHEM101, and BIO101”), grades received by the student for eachof the classes (“A, A−, B+, and A”), and a number of credits for eachclass (“4.0, 4.0, 4.0, 3.0”). The transcript also includes a number ofcredits for Winter 2019 (“15.0”), a grade point average (GPA) for Winter2019 (“3.8”), a cumulative amount of credits earned by the student(“56.0”), and a cumulative GPA for the student (“3.56”).

Turning now to FIG. 3, an exemplary graphical user interface (GUI) 300is illustrated. In an example, the GUI 300 may be presented as part ofthe graphical features 110 presented on the display 108 of the computingdevice 100. The GUI 300 includes a first pane 302 that displays adocument. As shown in FIG. 3, the document 200 is displayed in the firstpane 302.

The GUI 300 further includes a second pane 304. The second pane 304includes a plurality of fields 306-316. Each field in the plurality offields 306-316 is configured to receive input from a user that isindicative of defined criteria that may be used by the textualextraction application 106 in order to identify textual elements fromdocuments of the defined type of the document 200.

The plurality of fields 306-316 may include a chain length field 306, analignment type field 308, a tolerance field 310, a gather to next field312, a coordinate type field 314, and/or a comparator field 316. Thechain length field 306 enables a user to specify a number of textualelements that are to be extracted in a chain of textual elements. Thealignment type field 308 enables a user to specify alignments (e.g.,right-aligned, center-aligned, left-aligned, justified) that are to beutilized in extracting textual elements from a document. This adjuststhe spatial target for the identified text, affecting the score forrelated text found in this area of an analyzed document. The tolerancefield 310 enables a user to specify a level of relatedness of extractedtextual elements. Put another way, the tolerance field 310 specifies aratio of allowed differences between textual elements learned during atraining process of the computer-implemented model 120 and textualelements that are extracted during operation of the textual extractionapplication 106. The differences may pertain to edit distances, eitherin normal textual space, format representation, or canonical textualrepresentations. The differences may also pertain to word-semanticdistances. When the tolerance field 310 is set to a high value, thetextual extraction application 106 identifies a relatively greaternumber of less related textual elements. When the tolerance field 310 isset to a lower value, the textual extraction application 106 identifiesa relatively fewer number of more related textual elements. In anembodiment, the tolerance field 310 may be configured to control forspatial factors between textual elements. For instance, in theembodiment, a greater value in the tolerance field 310 will allow for agreater distance and/or a greater angle between textual elements ascompared to a lesser value in the tolerance field 310. The gather tonext field 312 enables a user to specify a number of textual elementsthat are to be gathered adjacent to a textual element as part of a chainof textual elements. The gather to next field 312 may allow for anabsolute number of words next to textual elements to be extracted, aswell as a number of words within a relative distance (e.g., 40% of pagewidth) between textual elements to be extracted, or a number of first orlast words between textual elements to be extracted. The coordinate typefield 314 enables a user to specify a coordinate system (e.g., cartesiancoordinates, polar coordinates) that is to be utilized by the textextraction application 106. The comparator field 316 enables the user tospecify a format of the defined criteria. For instance, the format mayinclude string, integer, or decimal.

In an embodiment, textual elements shown in the document displayed inthe first pane may be selected by a user. In an example, the GUI 300 mayreceive a selection of a first textual element 320 (“Name: Joe A.Smith”) and a second textual element 322 (“Cumulative GPA:3.56”) by theuser. As seen in FIG. 3, the first textual element 320 and the secondtextual element 322 may be marked with first visual indicia to indicatethe selection within the GUI 300. In another example, the GUI 300 mayreceive a selection of a third textual element 324 (“MATH101”) and afourth textual element 326 (“A”) by the user. The third textual element324 and the fourth textual element 326 may be marked with second visualindicia to indicate the selection within the GUI 300. The third textualelement 324 and the fourth textual element 326 may be within a distancerange 328, for instance, as specified by the tolerance field 310. Thetextual extraction application 106 may utilize the selections of thetextual elements 320-326 in generating the defined criteria. Forinstance, as “MATH101” and “A” have been selected, the defined criteriamay include textual elements that match an introductory level math classand a grade for the introductory level math class, even if the textualelements corresponding to “MATH101” and/or “A” vary from document todocument of the defined type of the document 200 (educationaltranscript). For instance, for a different education transcript in which“MATH101” is substituted with “MTH1001”, the textual extractionapplication 106 (through use of the computer-implemented model 120) mayidentify textual “MTH1001” as a textual element that is to be extracted.

The GUI 300 may include a train chain button 318. When the GUI receivesa selection of the train chain button 318, the textual extractionapplication 106 may generate the defined criteria based upon selectionsof textual elements within the document 200 displayed in the first pane302 as well as contents of the plurality of fields 306-316.

Turning now to FIG. 4, another view of the GUI 300 is illustrated. Asshown in FIG. 4, a second educational transcript is shown in the firstpane 302. Certain textual elements within the second educationaltranscript differ from the educational transcript shown in FIG. 3. Forinstance, the identifiers for the classes now comprise four digitsinstead of three as in FIG. 3. Additionally, positions of textualelements differ within the second transcript as compared to theeducational transcript shown in FIG. 3.

As shown in FIG. 4, a fifth textual element 402 (“Jane C. White”), asixth textual element 404 (“Cumulative GPA:3.80”), a seventh textualelement 406 (“MATH1001”), and an eighth textual element 408 (“A”) havebeen selected by the textual extraction application 106 in accordancewith the above-described processes and based upon the defined criteriadescribed above.

In an example, the textual extraction application 106 may select thefirst textual element 402 and the second textual element 404 based uponan angle 412 between the first textual element 402, the second textualelement 404, and an axis 410 despite the fact that the sixth textualelement 404 is located at a different position than the second textualelement shown in FIG. 3. In another example, the textual extractionapplication 106 may select the seventh textual element 406 and the eighttextual element 408 as the seventh textual element 406 and the eighttextual element 408 are within the distance range 328 described above.Notably, the textual extraction application 106 selects the seventhtextual element 406 and the eight textual element 408 despite the factthat positions of the seventh textual element 406 and the eight textualelement 408 are different from positions of the third textual element324 and the fourth textual element 326, respectively, and despite thefact that the identifiers for the classes comprise four digits insteadof three digits.

Referring now to FIG. 5, an updated view of the GUI 300 shown in FIG. 3is illustrated. As shown in FIG. 5, the GUI 300 includes an extractedtext field 502. The extracted text field 502 is configured to displaytextual elements that have been extracted from the document 200 (or thatwill be extracted from the document 200) shown in the first pane 302based upon the defined criteria and output of the computer-implementedmodel 120. For instance, as shown in FIG. 5, the extracted text field502 may display an identifier for each course in the document 200 and agrade received by the student for each course.

With reference now to FIG. 6, a view 600 of a plurality of textualelements 602-630 in a document is illustrated. The textual extractionapplication 106 described above may determine contextually relevantrelationships between the plurality of textual elements 602-630 usingthe processes described above. In the view 600, textual elements in theplurality of textual elements 602-630 that have relevant relationshipswith one another are indicated by arrows. For instance, the textualextraction application 106 has determined that the textual element 612(“brown”) and the textual element 614 (“fox”) have a contextuallyrelevant relationship.

FIGS. 7 and 8 illustrate exemplary methodologies relating to extractionof textual elements from computer-readable text of documents. While themethodologies are shown and described as being a series of acts that areperformed in a sequence, it is to be understood and appreciated that themethodologies are not limited by the order of the sequence. For example,some acts can occur in a different order than what is described herein.In addition, an act can occur concurrently with another act. Further, insome instances, not all acts may be required to implement a methodologydescribed herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

Referring now to FIG. 7, a methodology 700 executed by a computingdevice for generating a computer-implemented model is illustrated. Themethodology 700 begins at 702, and at 704, the computing device accessesa plurality of documents of a defined type from a data store. Eachdocument in the plurality of documents comprises computer-readable textand a layout that defines positions of the computer-readable text withina two-dimensional area represented by each document. At least somedocuments in the plurality of documents have characteristics that vary.The characteristics may include varying portions of computer-readabletext, varying positions of the computer-readable text within thedocuments, varying typographical emphasis of portions of thecomputer-readable text, varying areas of the plurality of documents,varying lengths and widths of the plurality of documents, varying fonttypes of the portions of the computer-readable text, and/or varying fontsizes of the portions of the computer-readable text.

At 706, the computing device generates a computer-implemented modelbased upon the plurality of documents. The computer-implemented model isconfigured to take, as input, at least some of the characteristicsdescribed above from a document of the defined type. Thecomputer-implemented model outputs, based upon the input, a plurality oftextual elements in the document and scores assigned to the plurality oftextual elements. A score in the scores is indicative of a likelihoodthat at least one textual element in the plurality of textual elementsrepresents relevant content in the document based upon defined criteriafor the defined type. The methodology 700 concludes at 708.

Turning now to FIG. 8, a methodology 800 executed by a computing devicefor extracting textual elements from computer-readable text of adocument is illustrated. The methodology 800 begins at 802, and at 804,the computing device receives a document comprising computer-readabletext and a layout. The layout defines positions of the computer-readabletext within a two-dimensional area represented by the document.

At 806, the computing device identifies at least one textual element inthe computer-readable text of the document based upon spatial factorsbetween portions of the computer-readable text and contextualrelationships between the portions of the computer-readable text. Assuch, the computing device provides the computer-readable text and thepositions of the computer-readable text within the document as input toat least one computer-implemented model. The at least onecomputer-implemented model outputs, based upon the input, a plurality oftextual element within the computer-readable text and scores assigned tothe textual elements. The at least one textual element is included inthe plurality of textual elements. The computing device identifies theat least textual element based on a score in the scores. The score isindicative of a likelihood that the at least one textual elementrepresents relevant content in the document based upon defined criteriafor a defined type of the document.

At 808, responsive to identifying the at least one textual element inthe computer-readable text, the computing device outputs the at leastone textual element. The methodology 800 concludes at 810.

Referring now to FIG. 9, a high-level illustration of an exemplarycomputing device 900 that can be used in accordance with the systems andmethodologies disclosed herein is illustrated. For instance, thecomputing device 900 may be used in a system that extracts textualelements from computer-readable text of a document. By way of anotherexample, the computing device 900 can be used in a system that displaystextual elements that have been extracted from computer-readable text ofa document. The computing device 900 may thus be or include thecomputing device 100. The computing device 900 includes at least oneprocessor 902 that executes instructions that are stored in a memory904. The instructions may be, for instance, instructions forimplementing functionality described as being carried out by one or morecomponents discussed above or instructions for implementing one or moreof the methods described above. The processor 902 may access the memory904 by way of a system bus 906. In addition to storing executableinstructions, the memory 904 may also store document images, documents,computer-implemented models, extracted textual elements, etc.

The computing device 900 additionally includes a data store 908 that isaccessible by the processor 902 by way of the system bus 906. The datastore 908 may include executable instructions, document images,documents, computer-implemented models, extracted textual elements, etc.The computing device 900 also includes an input interface 910 thatallows external devices to communicate with the computing device 900.For instance, the input interface 910 may be used to receiveinstructions from an external computer device, from a user, etc. Thecomputing device 900 also includes an output interface 912 thatinterfaces the computing device 900 with one or more external devices.For example, the computing device 900 may display text, images, etc. byway of the output interface 912.

It is contemplated that the external devices that communicate with thecomputing device 900 via the input interface 910 and the outputinterface 912 can be included in an environment that providessubstantially any type of user interface with which a user can interact.Examples of user interface types include graphical user interfaces,natural user interfaces, and so forth. For instance, a graphical userinterface may accept input from a user employing input device(s) such asa keyboard, mouse, remote control, or the like and provide output on anoutput device such as a display. Further, a natural user interface mayenable a user to interact with the computing device 900 in a manner freefrom constraints imposed by input devices such as keyboards, mice,remote controls, and the like. Rather, a natural user interface can relyon speech recognition, touch and stylus recognition, gesture recognitionboth on screen and adjacent to the screen, air gestures, head and eyetracking, voice and speech, vision, touch, gestures, machineintelligence, and so forth.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 900 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 900.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and blu-ray disc (BD), where disks usually reproducedata magnetically and discs usually reproduce data optically withlasers. Further, a propagated signal is not included within the scope ofcomputer-readable storage media. Computer-readable media also includescommunication media including any medium that facilitates transfer of acomputer program from one place to another. A connection, for instance,can be a communication medium. For example, if the software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio and microwave are includedin the definition of communication medium. Combinations of the aboveshould also be included within the scope of computer-readable media.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Application-specific Integrated Circuits (ASICs),Application-specific Standard Products (ASSPs), System-on-a-chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the details description or the claims,such term is intended to be inclusive in a manner similar to the term“comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. A computing system, comprising: a processor; andmemory storing instructions that, when executed by the processor, causethe processor to perform acts comprising: obtaining an electronicdocument comprising a first textual element and a second textualelement, wherein the first textual element is located at a firstposition within the electronic document, wherein the second textualelement is located at a second position within the electronic document,and further wherein the electronic document has a type; obtainingdefined criteria for the type of the electronic document; computing ascore that is indicative of a likelihood that the first textual elementrepresents relevant content in the electronic document based upon: acontextual relationship between the first textual element and the secondtextual element; spatial factors between the first position of the firsttextual element and the second position of the second textual element;and the defined criteria; selecting at least one of the first textualelement or the second textual element based upon the score; andoutputting the at least one of the first textual element or the secondtextual element.
 2. The computing system of claim 1, wherein the spatialfactors include one or more of: a distance between the first positionand the second position in the electronic document; or an angle betweenthe first position, the second position, and an axis of the electronicdocument.
 3. The computing system of claim 1, wherein outputting the atleast one of the first textual element or the second textual elementcomprises storing the at least one of the first textual element or thesecond textual element in a database.
 4. The computing system of claim1, the acts further comprising: prior to computing the score:determining the contextual relationship between the first textualelement and the second textual element; and computing the spatialfactors between the first position of the first textual element and thesecond position of the second textual element.
 5. The computing systemof claim 1, wherein the score is further based upon a typographicalemphasis of the first textual element.
 6. The computing system of claim1, wherein the type of the electronic document is one of: an educationaltranscript; an invoice; a medical record; a personnel record; or ataxation form.
 7. The computing system of claim 1, wherein the firsttextual element includes one or more of: at least one letter; at leastone number; at least one punctuation symbol; or at least onemathematical symbol.
 8. The computing system of claim 1, wherein thescore is further based upon a font size of the first textual element. 9.The computing system of claim 1, wherein the first textual element andthe second textual element are comprised by a table within theelectronic document.
 10. A method executed by a processor of a computingsystem, the method comprising: obtaining an electronic documentcomprising a first textual element and a second textual element, whereinthe first textual element is located at a first position within theelectronic document, wherein the second textual element is located at asecond position within the electronic document, and further wherein theelectronic document has a type; obtaining defined criteria for the typeof the electronic document; determining a contextual relationshipbetween the first textual element and the second textual element;computing a spatial factor between the first position of the firsttextual element and the second position of the second textual element;computing a score that is indicative of a likelihood that the firsttextual element represents relevant content in the electronic documentbased upon: the contextual relationship between the first textualelement and the second textual element; the spatial factor between thefirst position of the first textual element and the second position ofthe second textual element; and the defined criteria; selecting at leastone of the first textual element or the second textual element basedupon the score; and outputting the at least one of the first textualelement or the second textual element.
 11. The method of claim 10,wherein the contextual relationship is one of: source to object; objectto use; person to location; whole to part; or type to subtype.
 12. Themethod of claim 10, wherein the electronic document is generated byapplying an optical character recognition (OCR) process on an image of aphysical version of a document.
 13. The method of claim 10, whereinoutputting the at least one of the first textual element or the secondtextual element comprises presenting the at least one of the firsttextual element or the second textual element on a display.
 14. Themethod of claim 10, further comprising: prior to obtaining the definedcriteria, determining the type of the electronic document.
 15. Themethod of claim 10, wherein the defined criteria are received as inputfrom a user.
 16. The method of claim 10, wherein the contextualrelationship between the first textual element and the second textualelement is determined by way of a computer-implemented neural network.17. The method of claim 10, wherein the electronic document is asemi-structured document.
 18. A non-transitory computer-readable storagemedium comprising instructions that, when executed by a processor of acomputing system, cause the processor to perform acts comprising:obtaining an electronic document comprising a first textual element anda second textual element, wherein the first textual element is locatedat a first position within the electronic document, wherein the secondtextual element is located at a second position within the electronicdocument, and further wherein the electronic document has a type;obtaining defined criteria for the type of the electronic document;computing a first score that is indicative of a likelihood that thefirst textual element represents relevant content in the electronicdocument based upon: a first contextual relationship between the firsttextual element and the second textual element; first spatial factorsbetween the first position of the first textual element and the secondposition of the second textual element; and the defined criteria;computing a second score that is indicative of a likelihood that thesecond textual element represents the relevant content in the electronicdocument based upon: a second contextual relationship between the secondtextual element and the first textual element; second spatial factorsbetween the second position of the second textual element and the firstposition of the first textual element; and the defined criteria;selecting at least one of the first textual element or the secondtextual element based upon the first score and the second score; andoutputting the at least one of the first textual element or the secondtextual element.
 19. The non-transitory computer-readable storage mediumof claim 18, the acts further comprising: prior to computing the firstscore: determining the first contextual relationship between the firsttextual element and the second textual element; and computing the firstspatial factors between the first position of the first textual elementand the second position of the second textual element; and prior tocomputing the second score: determining the second contextualrelationship between the second textual element and the first textualelement; and computing the second spatial factors between the secondposition of the second textual element and the first position of thefirst textual element.
 20. The non-transitory computer-readable storagemedium of claim 18, wherein the first spatial factors are different thanthe second spatial factors.